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Microsoft COCO [2] is a new annotated database in 
computer vision consisting of more than 200.000 images. 
There are currently more than one million annotated objects 
from 80 categories, with fully segmented masks. With re¬ 
spect to Pascal [1], the previous available dataset with se¬ 
mantic segmentation annotations, COCO has four times the 
number of categories and two orders of magnitude more im¬ 
ages and annotated objects. 

In this context, the challenges for object detection in 
COCO have recently been presented. In a nutshell, compet¬ 
ing methods should provide a list of detections on each im¬ 
age in the form of bounding boxes or segmentation masks. 
Each detection should have associated one of the 80 cate¬ 
gories and a confidence score. 

These new challenges on a new dataset are unchartered 
territory for researchers used to work on Pascal for some 
years. This work’s main aim is to set a reference point in 
the challenges to get a first grasp of which results to expect. 

To do so, we provide Oracle MCG, a hypothetical detec¬ 
tor consisting of an oracle picking the best object proposal 
of a state-of-the-art technique on all annotated objects. This 
result could be understood as the upper-bound result one 
could get by restricting themselves to selecting object pro¬ 
posals without any further refinement. 

In particular, we use the publicly-available pre-computed 
results given by MCG [3] on the validation set of COCO 
(also available for the test set). On average, this method 
produces 5075 proposals per image. We then overlap all 
proposals on each image against all annotated objects and 
pick the best one for each of them. Each of these best pro¬ 
posals is associated to the annotated category and assigned 
a constant score of 1. 

The results obtained, as given by the COCO official eval¬ 
uation software, are shown in Table 1 for bounding boxes 
and in Table 2 for segmentation masks. The highlighted 
row is the one that will define the ranking of the competi¬ 
tion. Below some observations about the results. 

• COCO detection is a very challenging competition, 
given that an oracle selecting the best out of a set of 
5000 proposals only reaches a performance around 0.3. 
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• Although detecting segmented masks is in theory 
much more challenging than detecting at bounding box 
level, the results on bounding boxes are not extremely 
better than for segmentation. 

• Small objects are very relevant in COCO. While the 
result for large objects is 0.533, this drops to 0.122 in 
small objects and goes up only to 0.292 overall. 

• The ranking measure gives a strong weight to the very 
precise delineation of objects by sweeping loU thresh¬ 
olds up to 0.95. The results for the threshold at 0.5 is 
0.605 compared to 0.292. 
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